Today I want to chat a little bit about survey analysis. Recently I’ve heard a few stories that disturbed me. One in particular resulted in what I would consider falsifying data. A large organization had hired an outside consultant to conduct some survey work. In the report from the consultant, the response category of “other” had disappeared from a question and after a careful look at the raw data, the responses of “other” had been split up and lumped into the other existing categories. While some people that responded “other” might have actually provided a response that fit in one of the original categories….they still selected the response of “other”.
Is this cleaning the data or is this mishandling the data?
It’s one thing to clean the data for incomplete responses or responses that did not record properly but it’s another to move categories around after the survey had closed. To me, this indicates poor survey design. This isn’t always the case, but it certainly happens.
Now, not to say that we shouldn’t use the category of “other” or that well designed surveys don’t have this category. Quite the opposite, actually. There are times when you absolutely should be using “other” to allow participants to describe their particular experience or when a program has several parts that may fluctuate and you won’t be able to adjust the survey for last minute additions or issues. “Other” can be a very important and informative category.
So let’s pretend that we’ve created a survey for a local college’s orientation and we do need to include the category of “other”. (Note: The category “other” should not be used when pertaining to gender identity. People aren’t “other”. Don’t make them feel that they are. For more information click here.) We have distributed the survey and collected responses from about 50 participants. One question that used “other” was our question on which part of college orientation students liked best.
Looking at the data, some students picked “other”. (And a lot of students seemed to enjoy the talk on financial aid which is a little ridiculous but hey, it’s not real data) Our reaction is often to just ignore this category…but let’s stop for a second and look at it. If we look at the data for more than 10 seconds we will actually see that “other” had more responses than one of our actual categories. This…might indicate a flaw in our survey. So let’s look at the actual data. We should have had a space for students to fill in what they meant by “other”. If we didn’t, then we really have failed to represent the people we are surveying.
From reading the responses we can see that several students enjoyed meeting the faculty members. Maybe we should have had that as a category. We can also see that one student did like the campus tour which was an option but they had other thoughts on it, so they wanted to share those. That information is still important and that response should not be moved to be grouped with “campus tour” in my opinion. The respondent chose “other” and we need to respect their choice. To change the category is going to far with “data cleaning”.
By looking at what falls under the “other” umbrella we can get a better sense of what we need to ask in our future surveys and what a reasonable number of our respondents were saying. This was an oversimplified example, but I think it translates. I’ve seen so many rich and important points come from the category of “other”. This category needs to be examined, reported, and considered in future survey design. Too often “other” is swept under the rug. Don’t let that happen in your survey design or reporting!
Do you overlook “other”? Have you seen other surveys poorly handling this category? What’s the best information you’ve found hidden in an “other” response? Let me know in the comments!
Projects half Done 
I think there’s a happy medium between what you consider poor survey practice and what you propose here. I would argue putting the “meeting other people” as a column in your chart and, in the narrative, explain that many people chose “other” and wrote that they met other people. Or perhaps having a stacked bar chart for “other” with direct labels showing that majority of “other” was “meeting other people.”
Also, personally I would still clump the other response about the campus tour with the other campus tour questions. I think the answer to the question shows they are not really answering the question appropriately; perhaps there was no section for them to convey their recommendations for improving the orientation or perhaps a multiple choice option would have improved responses. Regardless, they are answering that the campus tour -was- the most useful part -but- they have recommendations for improving it (separate question). I believe most of my clients would find it confusing that they were not put into the “campus tour” category.
Dana,
I can see your perspective there and I agree that the respondent was most likely not answering the question as we would want them to, which again shows we may want to design a better question! Great thoughts.